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Abstract 

We study an early effort amid the Covid-19 pandemic to develop new approaches to virtually 
serving students, supporting teachers, and promoting equity. This five-week, largely 
synchronous, summer program served 11,769 rising 4-9" graders. “Mentor teachers” provided 
PD and videos of themselves teaching daily lessons to “partner teachers” across the country. We 
interviewed a representative sample of teachers and analyzed educator, parent, and student 
surveys. Stakeholders perceived that students made academic improvements, and the content was 
rigorous, relevant, and engaging. Teachers felt their teaching improved and appreciated receiving 
adaptable curricular materials. Participants wanted more relevant math content, more 
differentiated development, and less asynchronous movement content. Findings highlight 
promising strategies for promoting online engagement and exploiting virtual learning to 
strengthen teacher development. 
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Motivation 

The Covid-19 pandemic created an unprecedented disruption to our nation’s school 
systems, leading to the near-universal closure of school buildings for the last three months of the 
2019-2020 school year. At that time, scholars projected dramatic learning loss and a sharp 
increase in educational inequality (Kuhfeld & Tarasawa, 2020; Kuhfeld et al., 2020). Yet the 
disruption also generated a range of efforts to use remote instruction to mitigate these effects. 
The best available evidence suggests in-person instruction is preferable to virtual learning for 
student engagement and achievement (Bueno, 2020; Gallagher & Cottingham, 2020), but few 
studies have emerged to date of efforts to innovate in the virtual learning space amid the 
pandemic. 

This study examines an early effort to develop new approaches to serving students and 
promoting educational equity via virtual learning. Specifically, we study the National Summer 
School Initiative (NSSJ), a virtual program run over five weeks in summer 2020 that served 
11,769 3" to 8" grade students across the nation with the goal of minimizing Covid-19 learning 
loss. The roughly 50 partner schools or networks serve student populations that, on average, are 
90% Black or Latinx and in which 79% qualify for subsidized meals (see Table 1). Roughly 43% 
of schools were operated by charter management organizations. The program was designed by 
current and former leaders of high-performing charter schools and a large school district home to 
a high-performing charter sector, in partnership with the nonprofit consultancy Bellwether 
Education Partners. 

Leaders recruited a group of “mentor teachers” they considered to be among the nation’s 
most talented educators to videotape themselves teaching each lesson to their own “fishbowl 


class” of students via Zoom. Mentor teachers were assigned “partner teachers” selected by the 


partner school or network, with each mentor working with all of the roughly 50 English language 
arts or math teachers at each grade level. Partner teachers received access to lesson plans and 
video of the mentor teacher’s class session before teaching the same lesson to students from their 
home school. They also received professional development from their mentor teacher. Beyond 
supporting students, the program aimed to prepare teachers for a possible virtual 2020-2021 
school year and ease the burden on schools of creating robust summer programming while they 
focused on operational challenges and reopening plans. 

We examine stakeholder perceptions of the program to inform the design of future 
interventions, including those that seek to exploit virtual learning amid disruptions to in-person 
instruction. Numerous studies estimate the extent of summer learning loss in pre-Covid times, 
and evaluate programs designed to support struggling students during vacation breaks (e.g., Kim 
& Quinn, 2013; Lauer et al., 2006; Pyne, Messner & Dee, 2020; Schueler, 2018). However, 
scholars have yet to examine such efforts either in virtual settings or amid near-universal school 
closings. 

For 2020-2021, a large share of districts nationwide remain reliant on virtual or hybrid 
instruction. Leaders revised the NSSI model into an initiative called Cadence Learning that 
allows schools, districts, networks, and learning pods to gain access during the school year. 
Preliminary data on actual—trather than projected—learning loss due to Covid-19-induced school 
closures shows students achieving at lower levels in Fall 2020 than in prior years, especially in 
math, though a lack of testing data for many students leaves uncertainty about the full scope of 
the decline (Kuhfeld et al., 2020). Therefore, it is essential for the field to learn from early efforts 


to implement high-quality virtual instruction and mitigate learning loss. 


The National Summer School Initiative 

The student day at NSSI ran about three hours and forty-five minutes and the teacher day 
extended for an additional 90 minutes of preparation and PD. There were three core academic 
classes: novel studies, close reading, and math stories. Novel studies focused on reading and 
discussing what leaders described as an “exceptional novel” and developing pleasure in 
analyzing texts. In close reading, students discussed and wrote about shorter selections of poetry, 
short fiction, and nonfiction. In math stories, students solved a problem of the day designed to be 
“real world” applicable not by using particular algorithms but by applying their existing 
knowledge of math to the context. Teachers then led students through a “discourse” in which 
students discussed how they solved the problem, with the goal of developing Common Core- 
aligned conceptual understanding. The theory of action was that lower-performing students 
would be able to contribute ideas about how to solve the problem while also seeing a peer 
student’s more sophisticated solution. More advanced students would in turn be challenged to 
think of more than one method for solving. Novel studies and math stories were held daily, while 
close reading was held three days a week. The other two days, students had an enrichment class 
that included self-directed educational activities and synchronous virtual science labs. A daily 
asynchronous movement and mindfulness class included yoga, fitness, and dance. 

Prior to the program, teachers attended a week-long virtual training by the Lavinia 
Group, an organization that has previously served several leading charter networks. During the 
program, teachers had a daily PD session focused on intellectual preparation for the next day’s 
lesson and analysis of student work. The program was operated not for profit and funded by 
philanthropists. Partners received the curriculum, training, and mentor teacher services free of 


charge. They had to pay partner teachers, for any technology students needed to participate in the 


program, and for a local administrative coach to serve as their point of contact. Leaders 


recommended 20-30 students per section with a maximum of 40. 


Research Questions 
We draw on survey data from teachers, students, parents, and administrative coaches, as well 
as interviews with partner and mentor teachers, to address the following questions: 
1) To what extent did students, teachers, and parents perceive that NSSI affected student 
academic skills, interests, and attitudes toward school and learning? 
2) How did NSSI participation affect teacher self-reported morale, self-efficacy, and 
development? 


3) What did participants perceive were strengths and weaknesses of the program? 


Methods 

To learn about stakeholder perceptions of NSSI, we interviewed partner and mentor 
teachers and analyzed internal survey data. For interviews, we identified a stratified sample of 60 
partner teachers from the total group of 513 and 12 mentor teachers from the full group of 15 
based on the teacher’s subject and grade and whether the teacher worked at a charter 
management organization (CMO) school. Our interview sample ultimately included 28 teachers 
(22 partner teachers and 6 mentor teachers) who responded to our request. Table 2 indicates that 
the interview samples were representative of the full populations of teachers on all dimensions 
for which we have data. 

A team of eight researchers conducted interviews via Zoom using a semi-structured 


protocol. Interviews lasted between 45 and 60 minutes. Teachers received a $50 gift card for 


their time. Interviews were transcribed and double-coded using Dedoose coding software. The 
130 primarily inductive codes fell into categories such as curriculum, diversity, feedback, 
leaders, mentor teachers, morale, operations, schedule, students, teacher collaboration, teacher 
development, teachers, lessons about virtual learning, and workload. We analyzed the resulting 
coding for major themes summarized below. Protocol and codes are provided in the Appendix. 
We also analyzed data from surveys conducted by the NSSI team in weeks one, three, 
and five of the program. Our focus is primarily on the week five, end-of-program results to 
assess perceptions based on the full program. Surveys were completed by administrative coaches 
who served as the primary school- or network-based contact at each partner school (n=42), 
mentor teachers (n=7), partner teachers (n=188), students (n=2,484), and parents (n=892). Table 
2 shows that there are no statistically significant differences between the mentor survey sample 
and the full population of mentor teachers on observable characteristics. The partner teacher 
survey sample is generally representative but more likely to be missing data on the grade level 
taught (11% for the survey sample vs. 5% for the full population) and less likely to teach at a 
CMO (44% vs. 56%). We do not have data on the full population of students or parents to assess 
representativeness. All survey findings should be interpreted with caution as they may not reflect 


student, parent, and teacher opinions more broadly. 


Findings 
Stakeholders perceived that students made academic improvements. Partner and mentor 
teachers overwhelmingly perceived that students benefitted academically from participating in 


NSSI, although several acknowledged that there were limits to what could be accomplished in 


five weeks. This student learning theme came up in 23 of our 28 interviewees, and was tied for 
the most frequently used code. 
One ELA partner teacher described progress her students made with close reading and 
drawing connections between texts and real life: 
T honestly felt like all of my kids, they improved... The idea that you can read a poem, and 
it could have a literal meaning and a deeper meaning... when one picks up an article to 
read it, you're not just reading to get done and look for your teacher to ask you some very 
basic questions, but that this article is supposed to make you think about life... things that 
you're experiencing today. That skill was not even there and I saw them develop that. And 
most of them told me... I read differently now. And I pay attention to what I read. And I'm 
going to be thinking about what I read and what it means for my life. —Partner Teacher 1 
Math teachers similarly indicated that students at a variety of skill levels were able to 
become more flexible mathematicians: 
My students benefited a ton through this program. Even my students that may have 
caught on to the concepts quicker... they were more flexible... when they finished, they ... 
would try a second or third method because they had learned different ways of doing it 
from the discourse the day before. Even my kids who might not have had access to 
multiplication or division or are not as strong with those facts, they would then feel more 
confident with answering the questions because... even though they can’t do the quicker 
and most efficient method, they still were able to pull out other methods that they felt 
more confident in. —Partner Teacher 2 
Survey results presented in Tables 3 and 4 echo these themes. Among partner teachers 
responding to the end-of-program survey, large majorities agreed that students improved their 
academic skills (82%), gained confidence in their academic abilities (83%), and became more 
interested in school and learning (72%) due to NSSI. Mentor teacher survey respondents 
unanimously agreed that students gained academic skills, confidence, and interest in learning as 
a result of NSSI. Eighty-six percent of both partner and mentor teachers indicated NSSI 


improved their perceptions of virtual learning, although some interviewees were quick to point 


out that certain things are either not possible or not optimal in virtual settings. 


A majority of student survey respondents agreed that they grew as readers (81%) and 
mathematicians (75%), became more confident in their reading (68%) and math (65%) skills, 
and became more interested in school and learning (54%) during summer school, at least among 
those who completed the end-of-program survey (see Table 5). Parent responses (see appendix) 
exhibit a similar pattern. Most agreed their children improved in reading (75%) and math (77%), 
gained confidence in their academic abilities (75%), became more interested in learning (71%), 
and discovered a new interest (64%) due to summer school. 

Stakeholders perceived that the content was rigorous, relevant, and engaging— 
especially in ELA. Teachers told us that a key program strength was the content covered by the 
curriculum. Themes related to the quality of the curriculum were among the most frequently 
mentioned in our interviews. More than twenty interviewees emphasized that the curriculum was 
rigorous, culturally relevant, and engaging. 

Teachers were especially effusive about the novel choices, describing them as high- 
quality texts that elicited significant student investment. Teachers emphasized that, while 
engaging content is always important, this is especially critical in virtual settings where 
promoting engagement can be challenging. One ELA mentor explained: 

In an ELA classroom, the text is the most important thing. That's your starting point. You 

need to pick something that students are going to want to talk about. Being online... you 

need to pick a short text... not only do you want to choose content that's going to be 
really engaging, challenging for students, but really relevant and something that they can 
latch on to, it also needs to be something that they can feel like we're moving fairly 
quickly through this versus oh we’re slogging through... The curriculum for [my grade’s] 

ELA class was amazing. It was an incredibly relevant and poignant text. Students were 

really invested... there is a particular scene in the book where you find out that two 

characters have died ... as soon as they read that portion, [the partner teacher’s] class 
was kind of silent. And then a student who really hadn't spoken for most of the summer 
unmutes himself and goes, ‘Really? ... Both of them had to die?’ and then hits mute 


again, and she could just tell that they were so upset... so affected by what happened... 
We want students to feel that invested and that connected. —Mentor Teacher I 


Several teachers emphasized the cultural relevance of the ELA curriculum and ways the 
texts allowed students to draw connections to current events or their own lives. Teachers also 
highlighted that the novels and close reading texts were thoughtfully paired to reinforce 
concepts. One partner teacher put it this way: 


The novel we read basically was about oppression and a girl fleeing from her native land 
to the United States. That opened up many different conversations because we know 
what's going on in our world today, as far as from Covid, to black and brown people 
being racially profiled, racial injustice, all the different things. I feel like us reading that 
novel—and it was a 12-year-old girl relating to the students—this girl had times where 
she was hiding in a closet, not able to go anywhere. Students were like, look I'm indoors 
because of Covid. There were so many connections that I feel were so intentional. And I 
feel like the students recognized that which helped spark that interest... Also, there were 
nonfiction articles we read, and related to the novel... And when students know things 
are put together, like if they know the why behind things, they're wanting to continue to 
investigate and learn, but if they don't see the connection, ... it's more like... why am I 
learning about this? ... At first though, it was a mystery. We started out with the 
nonfiction piece, and they were like, whoa, this is some deep stuff. Wow... I just don't 
understand. And then when I started reading the novel, they're like, oh, that's who that 
leader was? Oh my gosh, the leader! So it's like we gave them a suitcase filled with 
information. And once they got to the to the novel, they're like, oh, I get why we packed 
that in our suitcase, they started to make the connection, and it got them to the greater 
theme in the end. —Partner Teacher 3 


Teachers further suggested that the curriculum expanded students’ knowledge and 
awareness of global historical events: 


A fairly straightforward strength [of the] program is they started with really good 
books... The book we read was about the partition of India, this super powerful and 
important moment in history in a beautifully told book, kids learned a lot about history, 
kids discussed Gandhi's ideas. I don't know, what do you want your own kids to be 
doing? Probably reading a really good book, discussing important ideas about the world 
...vou want books to be both mirrors for kids that they can see their own culture affirmed 
and honored, but you also want books that can be windows for kids and help them see 
into new experiences that are different from what they know. And I thought the books that 
they chose did a pretty great job of providing a little bit of both. —Mentor Teacher 2 


A large majority (84%) of partner teacher survey respondents agreed that the curricular 
materials provided by NSSI were strong. Students gave additional indications that they found the 


program engaging, with 77% reporting that the energy in their online classrooms was positive 


and 65% indicating they were happy to be in summer school. In sum, stakeholders perceived that 


the ELA curricular content and novel selection was a key program strength. 


Stakeholders found the math pedagogy accessible and rigorous but believed the math 


content could have been more culturally relevant. Teachers told our interviewers that the 
approach to math instruction was engaging and rigorous for students at a variety of skill levels. 
The focus on developing flexibility and multiple methods of answering the same problem 


allowed lower-performing students to engage with the material by finding their own way to 


solve. It allowed higher-performing students to continue engaging even after they had discovered 


their first method of solving. One math partner teacher shared: 


The rigor level was, for the most part, ‘low floor high ceiling, ’... any kid can access it, 
and then they all allowed for multiple ways of solving which is really what made the 
discussion and the math really rich because even if this problem seems kind of easy on 
face value... there were so many ways of representing all this stuff, my high [performing] 
students weren't bored the whole summer... If you are coming in at a lower level, you can 
still access the problem. If you are coming in very strong in math already, there's still 
more you can do to make your work even better and to build your flexibility in solving 
problems. Especially because there was no order or progression in the way that they 
were presented. It could be anything from the whole year any day and that in itself was 
pretty rigorous. —Partner Teacher 4 


An 81% majority of student survey respondents agreed that they learned new strategies to 


solve math problems, and 72% agreed they became increasingly comfortable solving math 
problems in more than one way. Teachers also argued that the discourse approach to 
mathematics, encouraging students to talk through their reasoning and ask questions of one 


another, fostered high levels of engagement. One partner teacher explained: 


Because the conversations were being led by [the students], I think it was just way more 
enjoyable for them as well. And it was so much more enjoyable for me, because I was 
like, ‘man, this is like, actual fun and the kids don't hate it.’ And the kids that would come 
and had no clue, if they didn't understand the question the day before, they were the ones 
that then came in, and were asking so many questions when other people would share 
their work. ‘Why did you do it this way? What is that?’ Usually, I'm used to those kids 
just sitting there silently, like, ‘I didn't get it, so I'm not going to participate’ but I think 
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the discourse opened up that opportunity. If you didn't get the question, you still had so 
much opportunity to participate... It was not like a normal summer school at all. — 
Partner Teacher 5 
By the end of the program, 63% of student survey respondents agreed they were more 
comfortable sharing ideas in math class. 
Despite these strengths, teachers indicated that they believed the math problems 
themselves could have been more relevant to increase engagement further. Interviewees 


99 66 


described the problems as “bland,” “boring,” “standard” and “not particularly innovative.” One 
mentor teacher described it this way: 

I love the teaching approach... but the actual problems we put in front of kids were very 

‘blah,’ to put it bluntly. There was a lack of cultural relevance and a lack of just like, ‘I'm 

a teenager or preteen and I want to do math that actually is interesting to me or sparks 

some kind of interest versus some random problem about someone selling lemonade.’ — 

Mentor Teacher 3 

Several teachers noted that at least one mentor teacher tried to infuse the math curriculum 
with greater cultural relevance. While fellow teachers appreciated the intentions behind this 
effort, they noted that one math problem she developed—with a set up related to the “three-fifths 
compromise” under which enslaved people were treated as less than a full person in allocating 
representation under the U.S. Constitution—backfired. Without introduction to put the problem 
in context, some partner teachers and parents found the exercise offensive. A handful of 
interviewees suggested that NSSI leaders, despite their overall anti-racist orientation, could have 
done more to address this incident head on. 

Stakeholders reported lower levels of student enthusiasm for the movement and 
mindfulness content. An important feature of NSSI was its emphasis on synchronous 


instruction. One exception were the movement and mindfulness classes which were pre-recorded 


and distributed to students for independent viewing. Administrators did not find this approach to 
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work especially well, with fully half of respondents disagreeing with the statement “movement 
and mindfulness was effective” (see Appendix Table A2). 

Teacher interviewees also indicated low levels of engagement with this content. One 
math partner teacher suggested this was due to the lack of synchronous interaction: 

The one thing they could kind of beef up a bit—but it was actually a great concept—was 

the movement and mindfulness. We were able to eliminate that block because our 

scholars did not respond to it. They really were not interested ... when it's all just a 

video, and they can't have any input, our scholars tend to zone out. —Partner Teacher 6 

Student survey respondents indicated low levels of engagement, with 27% by the end of 
the program indicating they had never attended in the past week. Open-ended survey responses 
suggest that the asynchronous nature of the content was to blame. One student wrote, “I would 
go more if it was live.” Others described the class as “boring” or needing more variety. 

Partner teachers perceived that the program improved their instruction. Overall, 
partner teachers felt that they improved their teaching as a result of participating in NSSI. 
Interviews revealed this was, in large part, due to access to the mentor teachers, who they 
generally perceived to be talented educators. The most common mechanism through which 
partner teachers suggested this occurred was by watching videos of mentor teachers leading 
classes through the same lessons partner teachers would teach four to five days later. Even more 
experienced teachers said this was helpful both in providing models for teaching the lesson and 
anticipating how students were likely to respond to particular parts of the curriculum. 
Representative comments from ELA teachers include: 

It was helpful—the mentor videos especially—to watch them and learn, okay, this is when 

she asks this question, and those transferable questions were really helpful, that was 

newer to me. So it was helpful to see how they would take a paragraph or part of the 
book and kind of break it down ... to see another teacher who's more experienced than 


me teach her students and where she would pause and what the key points were. —Partner 
Teacher 7 
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It was great to see someone else already roll out the lesson. In part because seeing other 
students’ responses helped me anticipate what direction my kids may or may not take. 
And then I think it's always helpful to watch other teachers teach. That's actually not 
something we have a ton of time or opportunity to do in a regular school setting... Those 
mentor teachers are now some of the people I've observed teaching the most, in my entire 
six years of teaching, because I got to see them do a full 45-minute lesson every day. 
Whereas really thinking about like any colleagues I've had over the past six years, I've 
never seen anyone do a complete 45-minute lesson, let alone every single day. Just 
having that experience of getting to observe another really excellent teacher, was just 
great. —Partner Teacher 8 


Survey data echoed these themes. Among partner teacher respondents, 80% agreed the “daily 
lesson videos and lesson flow documents provided by the mentor teacher are strong” and 79% 
agreed, “I am learning from my mentor teacher,” including 53% who strongly agreed. Among 
administrators, 67% agreed the “daily videos and lesson flow documents provided by mentor 
teachers are strong.” 

Some partner teachers indicated that access to mentors from across the country was 
particularly valuable for those whose home districts had a limited supply of highly effective 
teachers. One partner teacher explains how this was true for hard-to-staff subjects: 

Where I live, the math teachers I feel generally don't really understand the Common Core 

content. And I think there's a shortage of math teachers. So I think [NSSI] gave teachers 

a time to learn from people who really understand the content... and who really could 

teach them how to teach. —Partner Teacher 6 

Partner teachers also appreciated that mentors the same lesson before they did and 
informed them about what worked and what did not. Mentors indicated that teaching these 
lessons built their credibility with partner teachers and improved the PD they provided. One 
math mentor explained: 

I've done some whitewater kayaking in my in my life—and it's like the first run, right? 

You go out there, you get knocked over, you figure out where the eddies are, where the 

rocks are, and the currents, and then you come back and you say, ‘Look, I survived the 

thing and let me tell you all about it.’ So I think the teachers on the whole really 


appreciated that approach. I wasn't with them. I was one of them. I was just going four 
days earlier, trying to try things out, and some work, some didn't. I came back and 
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reported on it, showed them the video, and then they could learn in that way. And I would 

say you know never in the history of education has every second of every piece of 

instruction been recorded. Right? And that's what we what we accomplished... the 

promise for teacher development in that is really powerful.” —Mentor Teacher 4 

In terms of the skills partner teachers developed, interviewees indicated that the program 
helped them prepare to teach more effectively online in preparation for a virtual or hybrid school 
year. Many suggested that they went on to become leaders at their home schools, providing 
guidance and support to colleagues who were less experienced with virtual learning. A large 
majority (87%) of partner teacher survey respondents agreed with the statement, “I will be a 
better teacher in the 2020-21 school year because of teaching at NSSI,” including 57% who 
strongly agreed. 

Teachers said they improved their ability to promote student engagement, something they 
saw as a major challenge of virtual instruction. In fact, our code “virtual lessons — engagement is 
key” (indicating a takeaway about virtual learning) was among the most commonly applied 
codes across interviews. Partner teachers further argued that they improved on analyzing student 
work, giving feedback to students, featuring examples of student work to increase engagement, 
creating joyful online classrooms, and raising their expectations for what students could 
accomplish virtually. Others suggested that the program helped them realize that culturally 
relevant curriculum is important and possible to implement. 

Mentor teachers enjoyed the program and felt their own practice improved. Mentor 
teachers were some of our most enthusiastic interviewees, suggesting that they appreciated the 
opportunity to participate in the program—despite the heavy commitment it involved—and even 
that their own teaching practice improved as a result. Mentors especially appreciated the ability 


to develop relationships with and learn from a community of other excellent teachers across the 


country, as two teachers (one math and one ELA) explained: 
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The quality of people that were hired was unbelievable—a lot of really talented, smart, 
thoughtful, hardworking folks of a variety of different backgrounds really helped us have 
a pretty rich conversation about many topics, ranging from pedagogical approach to 
what does our organization stand for when we talk about fighting for racial equity ... The 
people component was really wonderful. —Mentor Teacher 3 


I really loved it... I would not say that it was just a walk in the park but one of the things 
that I really enjoyed was the chance to collaborate with other educators and to hear their 
perspectives from a number of different contexts... being able to learn from [a fellow 
mentor teacher] and the vast amount of experience that he has and being able to 
collaborate together. I feel like the professional relationships that I developed, I wouldn't 
trade those for anything. That alone made the summer worth it. -Mentor Teacher 1 


In addition to working with partner teachers and providing guidance based on their own 
efforts to implement the curriculum, mentor teachers appreciated having a role in creating the 
program with NSSI leaders and felt they were given substantial autonomy to experiment: 


What I loved about the organization was that it basically said, look, we have an idea of 
what we think the model is, but a big part of the model is hiring what we think are 
amazing teachers and letting them run with it... There was this kind of founders, 
innovation kind of spirit that was pervasive. There was a structure, and then within that 
structure, I felt like a musician who could riff: I could be like, ‘oh, I'm going to try this, 
I'm going to put on a costume, I'm going to say this thing, I'm gonna spend eight minutes 
on this or 12 minutes tomorrow. I felt a real sense of freedom and innovation in the 
model that was really powerful. And that innovation done in small ways over and over 
led to a better and better experience for the kids over time. And because of our size, 
relatively small, the sharing of those best practices across the teachers, you just saw 
things get adopted and shared really quickly. —Mentor Teacher 4 


Finally, mentor teachers appreciated that NSSI allowed them to expand their reach and 
have greater impact on students and teachers without leaving the classroom: 
I'm a classroom teacher, and [NSSI] provides a really big platform for a classroom 
teacher. This whole debate in my field of, well, if you teach in the classroom, you only get 
to teach this many kids but if you become the superintendent or the director of this or 
that, you teach thousands of kids, but you don't really teach them you do things that allow 
them to learn more. This is a way to stay in the classroom but then have way more kids be 
impacted by the stuff you're doing, which is unique. —Mentor Teacher 4 


Survey data was consistent with the interviews. Mentor teacher respondents unanimously 


agreed with the statement “I am happy that I am a mentor teacher this summer with NSSI,” 
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including 57% who strongly agreed, and also unanimously agreed that ““NSSI increased my 
enthusiasm about being a teacher.” Satisfaction appeared to increase over time, as we show in 
Appendix Table A5. Our interview data suggest that this was due to the heavy workload as the 
inaugural program launched as well as mid-program adjustments leaders made based on early 
feedback. 

Teachers appreciated receiving adaptable curricular materials. Another key program 
strength that partner teachers highlighted was the fact that NSSI provided them with what they 
perceived to be high-quality lessons rather than asking individual teachers to develop their own. 
This saved a significant amount of preparation time that they could instead use on other aspects 
of their teaching. However, they appreciated the flexibility they had to use these materials as 
they saw fit and to adapt them to their own teaching style and student needs. While some 
teachers actually played parts of the mentor teacher videos for their students, hit pause, ran the 
discussion, and then returned to the lesson, others simply watched the videos on their own in 
preparation for teaching their classes. As two partner teachers explained: 

I've taught summer school loads of times through those 31 years and most of the time I 

had to just create stuff—so being able to have access to high-quality materials really 

streamlined what we were doing at summer school. We were really able to focus on 
meeting kids where they were at and meeting their needs, and really providing some 

extra education. —Partner Teacher 9 

Not only did we have a very detailed and well thought out lesson plan to read, we also 

had a video of a person actually teaching this lesson. And so that just made preparation, 

I honestly don't know what more you could want... Teaching is hard and there’s not 

enough time... Just knowing how much time goes into crafting lessons and planning. If 

more teachers had all of that time to (1) really internalize the lessons, (2) really focus on 
student work, give feedback and adjust for the next day... That’s how you get the best 
outcomes for students is when the teachers really deeply know what they ’re teaching and 
where they’re going with it. And I still think the NSSI model is adaptable and leaves lots 
of room for people to be teachers in their own way. It’s not scripted... I could see the 


same lesson being carried out different ways at different schools or classrooms. —Partner 
Teacher 8 
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One of the ELA partner teachers (Partner Teacher 11) who taught a group of students at 
NSSI who he had not worked with in the past, when asked whether he was able to build a 
relationship with them, responded, “Totally. And part of that was because I wasn't spending so 
much time setting up lesson plans and worried about this and that. The fact that I knew certain 
things are already set up for me, it allowed me to spend time just getting to know the kids, their 
strengths, their weaknesses, their likes, and it just kind of flowed.” 

Teachers wanted the professional development to be more differentiated. Although 
partner teachers generally felt NSSI helped them improve, they had suggestions for how to 
improve the pre-program and daily professional development sessions. At the program’s start, 
teachers felt there was too much time built into the schedule for PD. Leaders responded by 
reducing the time—which was appreciated—but some teachers indicated that it was still too 
frequent and the placement in the schedule made it difficult to get the most of the PD because 
there was limited time between class and PD sessions. Partner and mentor teachers agreed 
expectations for PD attendance were also unclear. 

Furthermore, several teachers suggested that components of the PD could have been 
further differentiated by teacher experience level, particularly to increase the value for more 
experienced partner teachers. One ELA partner teacher explained: 

Everyone I worked with from my particular school, most all of us are veteran teachers. 

And I remember going through even the trainings and we were like saying this was not 

catered to teachers who've been core veteran teachers. This is catered to teachers who 

don't really have content knowledge. The questions that were being brought up by some 
teachers from other networks were questions about like, how do you do these simple 


things that are first year teacher questions. So all of us had similar feelings that this was 
not differentiated. —Partner Teacher 12 
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Mentor teachers we interviewed agreed that this was an area for improvement in terms of 
both differentiation and using the time in a less top-down way to engage the partner teachers 
more actively. One math mentor teacher argued: 

The areas of improvement would just be around differentiation for teachers in terms of 

how we supported them in supporting providers was more geared towards novice 

teachers, and not kind of like support for more veteran teachers, or at least space for 

collaboration for veteran teachers. —Mentor Teacher 5 

This was consistent with the survey data. Among administrators, only a bit more than half 
(54%) agreed that the “daily intellectual prep PD made my teachers stronger” and only 41% 
agreed that the “daily student work analysis made my teachers stronger.” Among the mentor 
teachers, 57% thought the intellectual prep was making teachers stronger, and only 29% agreed 
that the student work analysis time was making partner teachers stronger. Partner teachers were a 
bit more positive about these components, with 61% agreeing the intellectual prep was helping 
and 57% agreeing that the student work analysis time was helping. Unfortunately, we do not 


have data on teacher experience for the survey sample to test whether newer teachers were more 


satisfied with the PD. 


Implications 

This study of the National Summer School Initiative suggests that the program succeeded 
in providing a high-quality learning experience for thousands of students whose schooling had 
been interrupted by the Covid-19 pandemic while also providing professional growth 
opportunities for participating educators. Partner teachers and students who completed surveys 
reported marked growth in student learning and intellectual confidence. Participants perceived 
the curricular content to be high-quality, engaging, and culturally relevant, especially in ELA. 


Partner teachers valued the chance to work in sustained partnership with an expert mentor who 
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was teaching the same content, while the mentor teachers appreciated the unique opportunity to 
extend their reach. These same stakeholders perceived aspects of the program that could be 
improved: The daily math problems could have been more engaging. The asynchronous 
movement and mindfulness component was ineffective. Teachers wanted the professional 
development to be more differentiated according to their experience level. Even so, virtually all 
partner and mentor teachers reported that they were glad to have participated and would 
recommend it to a colleague. 

Our study is limited to self-reported perceptions and cannot speak definitively to changes 
in participating students’ academic performance. We also urge caution in drawing conclusions 
from our survey data since respondents may not be representative of all participants. 
Nonetheless, our findings on stakeholder perceptions suggest several lessons for the field at a 
time when virtual learning is widespread. First, teachers, students, and parent participants believe 
that learning is indeed possible in a virtual environment and generally felt that the program 
improved their perceptions of what can be accomplished with online education. Our results also 
highlight the challenge of promoting student engagement in virtual settings and seem to suggest 
the superiority of synchronous over asynchronous content for doing so. 

One advantage of virtual over in-person learning is that physical geography is not a 
constraint. At NSSI, talented mentor teachers from across the country were able to extend their 
reach, working with students and teachers outside of their home states through an online 
platform. Given the widespread use of virtual learning during the COVID-19 pandemic, 
educational leaders could capitalize on this feature of online learning to increase the access low- 


income students of color have to the highly effective teachers who are currently inequitably 
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distributed across schools (Boyd et al., 2011; Boyd, Lankford, Loeb & Wyckoff, 2005; 
Lankford, Loeb & Wyckoff, 2002). 

The study also speaks to teacher development. Mentor teacher videos provided powerful 
models for partner teachers. This could be accomplished within in-person settings but virtual 
learning facilitates the creation of videos that can be curated into libraries of teaching practice. In 
the future, online learning could facilitate more systematic observation of partner teachers by 
talented coaches than has historically been possible in traditional in-person schools. This would 
be consistent with other research demonstrating the positive effects of video-based observation 
on teacher perceptions of the evaluation process (Quinn, Kane, Greenberg & Thal, 2018; Kane, 
Blazar, Gehlbach, Greenberg, Quinn & Thal, 2020). 

Our study also reveals potential lessons for teaching and learning more generally, 
regardless of whether it be in a virtual or in-person setting. Our findings suggest that high- 
quality, rich, culturally relevant, timely content can be useful for generating student engagement. 
Pedagogy that allows students at multiple levels to access the content may also be valuable for 
promoting engagement in math. The results suggest that teachers perceive culturally relevant 
content to be helpful for student engagement, consistent with evidence on the causal effects of 
programs such as ethinic studies on engagement and achievement (Dee & Penner, 2017). 
However, our findings also illustrate how efforts to incorporate such content can backfire. This 
suggests the need for thoughtful design from the start, and that cultural relevance can be 
challenging to accomplish on the fly. 

Importantly, our findings strongly suggest that teachers need not reinvent the wheel when 
it comes to curriculum. Centralized efforts to provide high-quality—but still adaptable—lessons 


to teachers can save them valuable time that they can instead devote to other important tasks 
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such as internalizing lessons, providing feedback, and building relationships. This seems 
especially relevant in the times of coronavirus when educators are overwhelmed with the 
logistical challenges of physical distancing and developing virtual teaching skills but is probably 
a useful lesson for leaders regardless of the broader circumstances. Notably, our findings on 
teacher perceptions are consistent with existing evidence on the causal effects of providing 
curriculum for student achievement (Jackson & Makarin, 2018). 

We also see implications for teacher development within and beyond virtual learning 
environments. Mentor teachers’ experiences suggest that programs that differentiate teachers’ 
roles based on experience and effectiveness can generate enthusiasm among talented educators 
and multiply their impact. Partner teachers’ experiences suggest that sustained partnership with a 
talented mentor who shares materials and practices was perceived as a powerful instructional 
improvement strategy, with potential for both online and in-person instructional formats. Our 
results further suggest that teachers appreciate PD that is differentiated to their needs and levels 
of experience. Teachers also appreciate coaching that is delivered by someone who has 
previously taught the material and can therefore provide instructional guidance with credibility. 

Our study cannot speak to the effectiveness of virtual learning as a whole, nor does it 
provide a comparison between the efficacy of virtual versus in-person instruction. However, our 
findings do suggest promising practices and provide some room for optimism about what can be 


accomplished through virtual learning in times when in-person schooling is restricted. 
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Tables 


Table 1. Describing NSSI Schools Relative to All U.S. Schools 


NSSI Schools All U.S. Schools 

N of schools 66 106,687 
N of states 16 51 

Mean SD Min Max Mean SD Min Max 
Minimum grade 3.72 1.24 3.00 7.00 3.97 1.52 3.00 8.00 
Maximum grade 6.86 1.69 3.00 8.00 641 1.49 3.00 8.00 
Enrollment 500 551 36 4161 497 438 1 13789 
Asian 0.02 0.07 0.00 0.51 0.04 0.08 0.00 1.00 
Black 0.56 0.39 0.00 1.00 0.17 0.26 0.00 1.00 
Hispanic 0.34 0.33 0.00 0.99 0.21 0.26 0.00 1.00 
White 0.07 0.15 0.00 0.64 0.56 0.34 0.00 1.00 
Subsidized Lunch 0.79 0.17 0.10 0.98 0.54 0.26 0.01 1.00 
Gifted 0.00 0.01 0.00 0.04 0.05 0.08 0.00 1.00 
IEP 0.09 0.13 0.00 0.56 0.09 0.14 0.00 1.00 
AetievemercGueany 131°!) 26:04 3.04 2.41 
Achievement (se) 0.07 0.02 0.26 0.04 0.02 0.30 


Note: Data are drawn from the Stanford Education Data Archive Version 3.0, averaging 
at the school level across all available school years (2008-09 to 2015-16). Achievement 
is based on an average of math and ELA test score performance normed to be 
comparable across states. 


Table 2. Describing the NSSI Teacher Study Sample 


Mentor Teachers Partner Teachers 
All Survey Interview All Survey Interview 
Teachers Sample Sample Teachers Sample Sample 

N of teachers 15 12 - 513 188 22 
Female 53% 50% 50% - - 82% 
Grade 5.2 ae) 5.86 5:15 5.19 5.67 
Minimum grade 3 3 ) 3 3 3 
Maximum grade 8 8 8 8 8 8 
Grade missing 0% 0% 0% 5% 11%* 5% 
Multiple grades 20% 0% 0% 38% 39% 24% 
ELA 40% 50% 43% 56% 51% 48% 
Math 60% 50% 57% 34% 34% 38% 
Substitute 0% 0% 0% 1% 1% 0% 
CMO 67% 75% 83% 56% 44% **% 54% 
Years teaching - - 1a i - - 8.91 
Survey missing 20% - 0% 65% - 59% 
Survey rating - 4.44 (0.88) 4.72 (1.00) - 4.48 (0.69) 4.42 (0.83) 


Note: Statistical significance refers to differences between the sample and the full population of NSSI teachers (in cases where it is 
possible to test for such a difference). Partner Teacher Survey Sample represents those answering the third wave survey in week 5 of the 
program. Subject refers to the subject the teacher taught at NSSI. Grade represents the lowest grade taught in cases where a teacher taught 
multiple grades. Years of teaching experience refers to the total number of years the teacher reported serving as a teacher or administrator 
at any school prior to working at NSSI. CMO refers to whether the teacher teaches at a school that is part of a charter management 
organization during the regular school year. All values represent percentages unless units are otherwise specified. *p<.05, **p<.01, 
**ED< 001. 
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Table 3. Partner Teacher Endline Survey Results 


N of teachers 
N of networks/districts 
Mean 
How likely are you to recommend teaching at NSSI to other teachers? 8.47 
Strongly 


The daily intellectual prep PD time is making me a stronger teacher - 
The daily student work analysis time is making me a stronger teacher 

The curriculum and lesson materials provided by NSSI are strong 

The daily lesson videos and lesson flow documents provided by the mentor teacher are strong 
I am enjoying working in partnership with my mentor teacher 

I am learning from my mentor teacher 

I will be a better teacher in the 2020-21 school year because of teaching at NSSI 

I am happy that I am teaching summer school through NSSI 

NSSI increased my enthusiasm about being a teacher 

NSSI improved my perceptions of virtual learning 

My students gained confidence in their academic abilities as a result of NSSI 

My students became more interested in school and learning as a result of NSSI 


188 
36 
SD Min Max 
1.66 2 10 
Strongly 
Disagree Disagree Neutral Agree Agree 
10 25 34 26 
15 24 28 29 
2 12 32 53 
4 16 30 49 
3 17 25 54 
5 15 26 53 
3 11 29 57 
1 6 29 64 
4 18 36 41 
4 10 28 58 
| 17 53 30 
3 25 49 22 
1 54 2 


eco COO SO - —- =—WN & 


My students improved their academic skills as a result of NSSI 17 9 
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Table 4. Mentor Teacher Endline Survey Results 


N of teachers 7 
Mean SD Min Max 

How likely are you to recommend being a mentor teacher at NSSI to other teachers? 8.86 1.21 7.00 10.00 

Strongly Strongly 

Disagree Disagree Neutral Agree Agree 
The daily intellectual prep time with my partner teachers in making them stronger 0 14 29 57 0 
The student work analysis time with my partner teachers is making them stronger 0 14 57 29 0 
The resources from the Lavinia Group are strong 0 I4 43 43 0 
I am happy that I am a mentor teacher this summer with NSSI 0 0 0 43 57 
NSSI increased my enthusiasm about being a teacher 0 0 0 71 29 
NSSI improved my perceptions of virtual learning 0 0 14 29 57 
Students gained confidence in their academic abilities as a result of NSSI 0 0 0 71 29 
Students became more interested in school and learning as a result of NSSI 0 0 0 57 43 

0 0 


Students imp roved their academic skills as a result of NSSI 0 71 29 
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Table 5. Student Survey Endline Results 


N of students 2508 
N of schools 36 
Mean SD 
Grade 5.25 1.57 
Percent CMO 64.11 
Neither 
Strongly Agree nor Strongly 
Disagree Disagree Disagree Agree Agree 
I am happy that I am participating in summer school. - 7 24 27 38 
The teaching in summer school is strong. 1 3 13 26 57 
My teachers give me good feedback that helps me improve my work. 1 3 12 27 57 
The mentor teacher videos we watch in my Reading and Math classes are helpful to my learning. 6 7 23 29 36 
The energy in my online classrooms is positive. 2 4 17 28 49 
I enjoy participating in my movement and mindfulness (dance and yoga) class. 24 15 25 17 20 
This summer, I discovered something new that I am interested in. 12 9 22 24 34 
Iam now more interested in school than I was before summer school. 11 Il 26 21 30 
I became more interested in school and learning because of summer school 7 10 28 27 27 
I am more confident in my math skills than I was before summer school 5 8 23 27 37 
I am more confident in my reading skills than I was before summer school 4 6 22 31 37 
I have grown as a mathematician 2 3 21 as 31 
I have learned new strategies to solve math problems 3 3 13 43 38 
I am more comfortable sharing my ideas in math class 5 8 24 35 29 
I am more comfortable solving math problems in more than one way 3 4 21 41 31 
I have gained reading skills that help me enjoy a wider variety of texts 2 3 20 42 33 
I am more confident in using evidence to support my thinking in reading 2 3 17 4! 37 
I have grown as a reader 1 2 16 42 40 
I am more confident discussing texts 3 4 20 39 34 
Some of the 
How many days in the past week did you attend your movement and mindfulness class? Never days Everyday 
27 4! 32 
Mostofthe Almost 
I attended summer school A few times Sometimes time everyday Everyday 


3 3 6 25 64 


Appendix 


Table Al. List of codes applied to interview transcripts (sorted by frequency) 


N of N of 
Code Excerpts Interviewees 
Curriculum - culturally relevant sD 23 
Students - learned 45 23 
Virtual lessons - engagement is key 39 23 
Curriculum - good level of rigor 48 22 
Overall - interviewee suggestion for NSSI improvement 73 22 
Teacher development - models on video (improved my teaching) 60 ya) 
Curriculum - engaging for students DD 21 
Teacher development - PD participation decreased over time 26 21 
Teacher development - prepared me for virtual school year 47 20 
Mentor teachers - talented, helpful 48 19 
Teacher collaboration - other great teachers 38 19 
Teacher development - improved teaching 54 19 
Schedule - timing was good for students 23 18 
Teacher collaboration - across country 26 18 
Workload - just right 30 18 
Operations - provided materials (saved time) 40 LZ 
Students - enjoyed program 23 17 
Teacher development - promoting student engagement 30 17 


Teachers - tailored / adapted program 39 17 


Virtual lessons - learning online is possible 
Teacher development - too much PD time 
Diversity, equity, inclusion - positive 
Morale improved - feel more prepared 


Teacher development - lack of differentiation based on teacher 
experience 


Workload - too heavy 

Curriculum - not relevant 

Great quote! 

Schedule - needed improvement 

Students - not engaged 

Morale improved - enjoyed program 

Overall - good intentions 

Teacher development - mentors going first 

Virtual lessons - it can be fun/joyful 

Operations - technology challenges 

Teacher development - analyzing student work 
Curriculum - allowed for differentiation 

Diversity, equity, inclusion - negative 

Morale improved - created connection/ community 
Students - positive exposure to virtual learning 
Teacher development - PD had bad timing in schedule 
Virtual lessons - greater access is possible, geography is out the window 
Virtual lessons - meaningful content is key 
Curriculum - pacing just right 


30 
Da 
29 
21 


30 
17 
21 
17 
21 
18 
17 
17 
21 
15 
20 
15 
17 
20 
16 
13 
12 
13 
16 
10 
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Overall - familiar faces were helpful / lack of familiar faces was 
unhelpful 

Students - NSSI addressed COVID learning loss 

Teachers - had agency / role in co-creation 

Curriculum - not differentiated (for students of different levels) 
Curriculum - pacing too fast 

Operations - materials were well-organized 

Overall - "the math incident" 

Overall - ambitious program - scale, speed 

Teacher development - PD was more intensive than regular school 
Virtual lessons - celebrating student models motivated students 
Virtual lessons - student community is key 

Virtual lessons - technology access matters 

Curriculum - not rigorous 

Feedback - I needed more during NSSI 

Leaders - responsive to feedback 

Mentor teachers - partner teachers didn't play videos for students 
Operations - operations were disorganized and/or confusing 
Teacher collaboration - not enough 

Teacher development - giving feedback to students 

Teacher development - Lavinia Group was not valuable 
Teacher development - Lavinia Group was valuable 
Curriculum - math story approach was good 

Curriculum - not engaging 

Morale improved - gave me purpose 


Nnrn7A WOO OO 
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Teacher development - creating joyful classroom 

Teacher development - culturally relevant curriculum important / possible 
Teacher development - one-on-one was helpful 

Teacher development - training (pre-NSSI) was ineffective 
Virtual lessons - anonymity/privacy has advantages 

Feedback - worked well for mentor teachers (feedback to mentor 
teachers) 

Teacher development - increased teacher expectations for students 
Teacher development - mentors should have observed partner teachers 
Teacher development - using examples of student work 

Virtual lessons - breakout rooms are helpful 

Virtual lessons - synchronous is valuable 

Workload - heavy be startup 

Feedback - program needs a more thorough impact evaluation 
Fishbowl - did not select "camera ready" students 

Fishbowl - did select "camera ready" students 

Fishbowl - selected representative group 

Leaders - lack of diversity among program leaders 

Morale improved - lack of testing was enjoyable 

Operations - teaching assignment was off (e.g., grade, subject) 
Schedule - unclear 

Teacher development - PD expectations were unclear 

Workload - inequitable ELA/Math 

Curriculum - not comprehensive 

Feedback - unclear who to go to for support/feedback 
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Leaders - lack of anti-racist leadership 

Operations - materials were unclear 

Students - grades would increase motivation 

Students - students w/ disabilities needs not met 

Teacher collaboration - collab w/ other partner teachers not helpful 
Teacher development - not enough training (before program) 
Virtual lessons - everything takes longer 

Virtual lessons - less is more 

Curriculum - skipped around 

Fishbowl - encouraged to select "camera ready" students 
Fishbowl - selected kids from mentor teacher home school 
Fishbowl - selected kids not from mentor teacher home school 
Mentor teachers - not helpful (to partner teachers) 

Operations - materials never came or were delayed 
Operations - too long (too many weeks of summer) 
Operations - too many grades per teacher 

Overall - promoted equity / targeted student population 
Students - developed new interests at NSSI 

Students - did not learn much 

Teacher development - needed more time to analyze student work 
Teachers - lack of teacher voice 

Virtual lessons - expand teacher reach 

Virtual lessons - no differences in motivation than in-person 
Virtual lessons - some things are not possible 
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Virtual lessons - students upload work 
Workload - light 


Feedback - fewer surveys, more focus groups (partner to mentor teachers) 


Feedback - too much feedback from partner teachers 
Mentor teachers - too many from charter sector 

Morale hurt 

Morale improved - greater reach 

Teacher recruitment - branding was too "reform-y" 
Teachers - developed new interests 

Virtual lessons - being a fishbowl student was motivating 
Virtual lessons - developing independence 


Virtual lessons - instructional quality is more important than fancy tech 
Virtual lessons - addressing absenteeism (be students can watch video 
later) 


Virtual lessons - do we need regular in-person school? 
Virtual lessons - taking attendance is harder than in person 
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Table A2. Admin Coach Post-NSSI Survey Results (n=42) 
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Percent CMO 
N partner teachers - math 
N partner teachers - ELA 
N of students 
Grade 3 
Grade 4 
Grade 5 
Grade 6 
Grade 7 
Grade 8 
Total 
Class size 
Percent attended 75%+ days 
Grade 3 
Grade 4 
Grade 5 
Grade 6 
Grade 7 
Grade 8 
Hours per week spent observing 
<1 hour 
1-5 hours 


Mean SD Min Max 
42.86 0.5 

4.1 6.88 1 45 
5.05 7.01 1 45 
29.56 53.08 0 300 
27.18 52.27 0 300 
28.71 51.18 0 300 
32.84 49.77 0 300 
32.87 47.70 0 300 
12.87 18.01 0 68 
150.55 230.41 6 1500 
27.36 44.78 6 300 
38.25 35.70 0 92 
40.63 35.75 0 86 
50.42 37.18 0 100 
54.19 32.86 0 100 
50.38 31.72 0 95 
33.94 34.83 0 95 
14.63 
53.66 


6-10 hours 
11-15 hours 
Percent students with tech issues 
<10% 
10-25% 
25-50% 
50-75% 
75-100% 
Percent student engagement level 
Not at all engaged 
Slightly engaged 
Somewhat engaged 
Very engaged 
Extremely engaged 
Level of agreement (scale 1-5) 
Daily CCC was effective 
Movement & mindfulness was effective 
Daily intellectual prep PD made my teachers stronger 
Daily student work analysis made my teachers stronger 
Curriculum & lesson materials provided by NSSI are strong 
Daily videos & lesson flow documents provided by mentor teachers are strong 
NSSI improved my perceptions of virtual learning 
I am happy that I am teaching summer school through NSSI 
How likely are you to recommend teaching at NSSI to other teachers (scale 0-10) 


24.39 
Lad 


45.24 
Soe | 
9.52 
2.38 
7.14 


0.00 
4.76 
47.62 
47.62 
0.00 


3.98 
2.69 
3.41 
3.38 
4.12 
3.81 
3.60 
4.05 
7.54 


0.95 
1.16 
1.24 
1.10 
1.11 
1.06 
1.21 
0.93 
2.44 
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Table A3. Parent Survey Results 


Week | Week 3 Week 5 

N of parents 1147 1467 903 
N of schools 36 35 30 

Scale Mean SD Mean SD Mean SD 
Grade 3-8 499 149 498 150 5.13 149 
Percent CMO 66.1% 72% 63.4% 
I am satisfied with my child’s experience in the summer program run by the NSSI 1-5 436 O89 438 O90 441 O88 
How likely are you to recommend other students attend the summer school offered by the NSSI? 1-10 8.76 1.95 871 2.03 856 2.16 
My child became more interested in school and learning as a result of summer school 1-5 391 116 400 1.04 
My child gained confidence in his or her academic abilities as a result of summer school 1-5 401 1.09 412 0.95 
My child improved his or her math skills as a result of summer school 1-5 3.88 L115 413 0.94 
My child improved his or her reading skills as a result of summer school 1-5 3.99 «#=104 409 0.90 
Through summer school, my child discovered something new that he or she is interested in 1-5 3.82 11S 3.86 1.10 


NSSI improved my perceptions of virtual learning 1-5 3.99 1.10 416 0.99 
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Table A4. Partner Teacher Survey Results 


Week | Week 3 Week 5 
Mean SD Min Max Mean SD Min Max Mean SD Min Max 

N of teachers 209 327 188 

N of networks/districts 45 48 36 

The daily intellectual prep PD time is making me a stronger teacher 359 119 1 373 101 1.5 3.68 110 1 5 
The daily student work analysis time is making me a stronger teacher 3.46 1.23 1 5 3.56 109 1 § 3.64 116 1 5 
How likely are you to recommend teaching at NSSI to other teachers? 7.64 2.11 1 10 7.85 205 0 10 8.47 166 2 10 
The curriculum and lesson materials provided by NSSI are strong 424 091 1 5 421 087 1 5 432 089 1 5 
The daily lesson videos and lesson flow documents provided by the mentor teacher are strong 4.12 1.02 1 5 420 0.92 1 § 424 090 1 5 
I am enjoying working in partnership with my mentor teacher 421 0.97 1 5 423 0.93 I § 430 090 1 5 
I am learning from my mentor teacher 421 095 1 5 420 0.96 I 5 424 097 1 5 
I will be a better teacher in the 2020-21 school year because of teaching at NSSI 415 097 1 5 429 086 1 5 441 079 2 5 
I am happy that I am teaching summer school through NSSI 4.15 090 1 5 4.29 O88 | 5 455 067 2 5 
NSSI increased my enthusiasm about being a teacher 3.82 102 1 5 4.14 087 2 5 
NSSI improved my perceptions of virtual learning 442 084 1 5 438 O87 1 5 
My students gained confidence in their academic abilities as a result of NSSI 3.72 103 1 5 412 069 2 5 
My students became more interested in school and learning as a result of NSSI 402 062 3 § 3.90 0.77 2 #5 
My students improved their academic skills as a result of NSSI 4.00 068 2 5 410 0.70 2 = 5 
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Table A5. Mentor Teacher Survey Results 


Week | Week 3 Week 5 

Mean SD Min Max Mean SD Min Max Mean SD Min Max 
N of teachers 12 12 7 
The daily intellectual prep time with my partner teachers in making them stronger 3.50 1.24 1.00 5.00 3.08 1.00 1.00 4.00 3.43 0.79 2.00 4.00 
The student work analysis time with my partner teachers is making them stronger 3.17 1.27 1.00 5.00 3.25 0.97 1.00 4.00 3.14 0.69 2.00 4.00 
How likely are you to recommend being a mentor teacher at NSSI to other teachers? 6.67 3.60 1.00 10.00 7.75 2.18 3.00 10.00 8.86 1.21 7.00 10.00 
Summary rating of above questions (comparable across all weeks) 444 0.88 3.33 6.00 4.69 0.86 3.67 6.00 5.14 0.74 4.00 6.00 
The resources from the Lavinia Group are strong 3.58 1.08 2.00 5.00 - - - - 3.29 0.76 2.00 4.00 
I am happy that I am a mentor teacher this summer with NSSI 4.00 1.04 2.00 5.00 - - - - 4.57 0.53 4.00 5.00 
Students gained confidence in their academic abilities as a result of NSSI - - - - 3.50 0.80 2.00 5.00 4.29 0.49 4.00 5.00 
NSSI increased my enthusiasm about being a teacher - - - - 4.17 1.03 2.00 5.00 4.29 0.49 4.00 5.00 
Students became more interested in school and learning as a result of NSSI - - - - - - - - 443 0.53 4.00 5.00 
Students improved their academic skills as a result of NSSI - - - - - - - - 4.29 0.49 4.00 5.00 


NSSI improved m ceptions of virtual learnin - - - - - - - - 4.43 0.79 3.00 5.00 
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Table A6. Student Survey Results 


N of students 
N of schools 


Grade 

Percent CMO 

I am happy that I am participating in summer school. 

The teaching in summer school is strong. 

My teachers give me good feedback that helps me improve my work. 


The mentor teacher videos we watch in my Reading and Math classes are helpful to my learning. 


The energy in my online classrooms is positive. 
I enjoy participating in my movement and mindfulness (dance and yoga) class. 
This summer, I discovered something new that I am interested in. 
I am now more interested in school than I was before summer school. 
How many days in the past week did you attend your movement and mindfulness class? 
Never 
Some of the days 
Everyday 
I became more interested in school and learning because of summer school 
I am more confident in my math skills than I was before summer school 
I am more confident in my reading skills than I was before summer school 
I have grown as a mathematician 
I have learned new strategies to solve math problems 
I am more comfortable sharing my ideas in math class 
I have gained reading skills that help me enjoy a wider variety of texts 
I am more confident in using evidence to support my thinking in reading 
I am more comfortable solving math problems in more than one way 
I have grown as a reader 
I am more confident discussing texts 
I attended summer school 
A few times 
Sometimes 
Most of the time 
Almost everyday 


Week | 


Week 3 


4103 


45 
Mean 
5.05 
69.24% 
3.65 
4.24 
4.30 
3.89 
4.09 
2.93 
3.45 
3.34 


22.32% 
43.23% 
34.44% 


SD 
1.55 


1.18 
0.89 
0.94 
1.13 
1.03 
1.44 
1.38 
1.38 


Week 5 
2508 
36 
Mean SD 
S25 157 
64.11% 
3.8707 1.12 
433 0.91 
4.37 0.88 
3.83 1.16 
4.18 0.99 
2.93 1.43 
3.58 1.35 
3.48 1.33 
26.50% 
41.19% 
32.31% 
357 61.19 
3.84 1.15 
3.91 1.08 
3.94 0.92 
4.11 0.92 
3.75 1.10 
400 0.91 
40 0.91 
3.94 0.97 
4.16 0.85 
3.98 0.97 
2.73% 
3.08% 
5.80% 
24.55% 


Everyday 63.85% 


Interview Recruitment Email 


Subject Line: Participate in NSSI Study Interview? 


Dear FirstName LastName, 


My name is Isabelle Edwards and I am a member of a research team based at the University of 
Virginia conducting a study of the National Summer School Initiative (NSSD, led by Dr. Beth 
Schueler. You have been selected to participate in an interview so that our team can learn more 
about your experience with NSSI this summer. The goal is to better understand what worked 
well and what could be improved to inform future iterations of the program, as well as similar 
programs around the country. 


The interview would occur by via Zoom (either video or audio only — your choice) and take no 
more than one hour. We would provide a $50 gift card as a thank you for your time. Your 
answers to interview questions would be entirely confidential and would in no way jeopardize 
your relationship with NSSI. Responses would be analyzed by our research team not NSSI staff. 


If you’re willing to consider participating, please review the attached consent form and let me 
know if you have any questions. If and when you’re ready to, please sign electronically and 
return the form to indicate your willingness to participate and we will be back in touch to 
schedule the interview at a time that is convenient for you. We can do weekdays, evenings, or 
weekends, depending on your availability. We hope to complete all interviews in the next two 
weeks, so we hope to hear from you soon. 


Many thanks for your time and consideration, especially during these challenging times. 


Best, 
FirstName 


FirstName LastName 
Research Assistant 
NSSI Study Team 
University of Virginia 
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Partner Teacher Interview Protocol 


Introduction: Thank you again for taking the time to participate today, especially since | realize this 

is a challenging time for most people. A few quick reminders before we get started: 

e First, the goal of this interview is for me to learn about your experience with NSSI to both 
improve the NSSI program in the future and identify lessons for the field as a whole about virtual 
instruction and teaching and learning in general. 

e Your participation is voluntary. You’re welcome to skip questions or stop the interview at any 
time. Your answers will be confidential and analyzed by researchers not NSSI staff. We will not 
use your name in any reports. 

e Finally, I would like to audio record this interview so that I can focus on listening rather than 
taking notes. Do I have your permission to record? <If yes, hit “record’’> 

e Any questions for me before we get started? 


Interview Questions: 
1) ’'m hoping we can begin by you telling me a bit about yourself for context. Where are you located 
geographically and how long have you been a teacher? [How many years?] 


2) Tell me about your experience participating in NSSI. Overall, how did it go? 


3) How was NSSI similar or different from a typical month at your school? What were the biggest 
differences? 


4) [If there were differences] were those differences good or bad? In other words, should school be 
more like NSSI, or should NSSI be more like school? 


5) In your view, what were the key strengths—if any—of the NSSI program? 
6) What were the major weaknesses or areas for improvement—if any—of the program? 


7) Did students benefit from the program? Why or why not? If so, how much and in what ways? 
How could you tell whether students benefitted? 


8) To what extent—if at all—did NSSI affect your students’ attitudes toward school and learning? 
9) Tell me about the workload on your end. Was it too little, too much, just right? How so? 
10) Tell me about the professional development component. 


11) Do you think the PD helped you improve your teaching? Why or why not? What worked and 
what could be improved? If it was not helpful, what would have been more helpful? 


12) How often did you attend the PD? If not always, tell me a bit about why (and no judgement one 
way or the other!). 


13) How did the PD compare to the PD you typically receive at your school? 


14) How was the schedule for your students? What worked well and what could be improved? 
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15) How was the content of the curriculum for your students? 
16) How was the rigor and pacing of the curriculum for your students? 
17) How relevant was the program content given the diverse backgrounds of NSSI students? 


18) Tell me about the level of student engagement with the program. What, if anything, did you learn 
from NSSI about promoting student engagement in virtual learning settings? 


19) How did NSSI affect your perceptions of what is possible (or not possible) with virtual learning? 


20) COVID-19 has made this a tough time for many teachers. How, if at all, did participating in 
NSSI affect your overall morale and feelings about teaching? 


21) There are a lot of downsides to virtual learning but one of the upsides is that geography is out of 
the window. Tell me whether and how much that mattered here. How important was interacting with 


teachers from other parts of the country? 


22) Tell me more about the experience of watching the mentor teacher’s lessons. Was that useful for 
you or not so much? Was there anything you saw that you will take and apply in your own teaching? 


23) At NSSI, how much interaction and collaboration did you have with other teachers? How did that 
compare to the interaction and collaboration you have in a typical month of school? 


24) Overall, would you return to NSSI or a similar program in the future? Why or why not? 
25) Is there anything you think other teachers and school systems can learn or take away from NSSI? 


26) What else should I know about the program or your experience with NSSI? 
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Mentor Teacher Interview Protocol 


Introduction: Thank you again for taking the time to participate today, especially since I realize this 

is a challenging time for most people. A few quick reminders before we get started: 

e First, the goal of this interview is for me to learn about your experience with NSSI to both 
improve the NSSI program in the future and identify lessons for the field as a whole about virtual 
instruction and teaching and learning in general. 

e Your participation is voluntary. You are welcome to skip questions or stop the interview at any 
time. Your answers will be confidential and analyzed by researchers not NSSI staff. We will not 
use your name in any reports. 

e Finally, I would like to audio record this interview so that I can focus on listening rather than 
taking notes. Do I have your permission to record? <If yes, hit “record’’> 

e Any questions for me before we get started? 


Interview Questions: 
1) ’'m hoping we can begin by you telling me a bit about yourself for context. Where are you located 
geographically and how long have you been a teacher and/or administrator? [How many years?] 


2) Tell me about your experience participating in NSSI. Overall, how did it go? 


3) How was NSSI similar or different from a typical month at your school (or the most recent school 
where you’ve worked)? What were the biggest differences? 


4) [If there were differences] were those differences good or bad? In other words, should school be 
more like NSSI, or should NSSI be more like school? 


5) In your view, what were the key strengths—if any—of the NSSI program? 
6) What were the major weaknesses or areas for improvement—if any—of the program? 


7) Did students benefit from the program? Why or why not? If so, how much and in what ways? 
How could you tell whether students benefitted? 


8) To what extent—if at all—did NSSI affect your students’ attitudes toward school and learning? 
9) Tell me about the workload on your end. Was it too little, too much, just right? How so? 
10) Tell me about the mentoring component of the program. 


11) How often did partner teachers in your group attend the professional development sessions? 
What worked and what could be improved? 


12) Do you think the mentoring helped partner teachers in your group improve their instruction? 
Why or why not? If so, in what ways and how could you tell? 


13) How did the mentoring compare to the mentoring or professional development that typically 
occurs at your school (or the most recent school where you’ve worked)? 
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14) Tell me about the feedback you received during the program (from NSSI leaders, partner 
teachers, or others). Was it helpful? Why / why not? What worked and how could the feedback 
process be improved? 


15) Tell me about the process of recruiting “showcase” or “fishbowl” students in your group. What 
worked and what could be improved? 


16) How was the content, rigor, and pacing of the curriculum for the students? 
17) How relevant was the program content given the diverse backgrounds of NSSI students? 


18) Tell me about the level of student engagement with the program. What, if anything, did you learn 
from NSSI about promoting student engagement in virtual learning settings? 


19) How did NSSI affect your perceptions of what is possible (or not possible) with virtual learning? 


20) COVID-19 has made this a tough time for many teachers. How, if at all, did participating in 
NSSI affect your overall morale and feelings about teaching? 


21) There are a lot of downsides to virtual learning but one of the upsides is that geography is out of 
the window. Tell me whether and how much that mattered here. How important was interacting with 


teachers from other parts of the country? 


22) At NSSI, how much interaction and collaboration did you have with other educators? How did 
that compare to the interaction and collaboration you have in a typical month of school? 


23) Overall, would you return to NSSI or a similar program in the future? Why or why not? 
24) Is there anything other teachers and school systems can learn or take away from NSSI? 


25) What else should I know about the program or your experience with NSSI? 


